Capsule neural networks

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network that is configured to receive a network input and to generate a network output for the network input. The neural network comprises a plurality of layers arranged in a sequence, including a plurality of capsule layers. Each particular capsule in a particular capsule layer is configured to receive respective inputs including: (i) outputs generated by capsules of a previous capsule layer that is before the particular capsule layer in the sequence, and (ii) final routing factors between capsules of the previous capsule layer and the particular capsule, wherein the final routing factors are generated by a routing subsystem. Each particular capsule in the particular capsule layer is configured to determine a particular capsule output based on the received inputs, wherein the particular capsule output is of dimension greater than one.

BACKGROUND

This specification relates to processing data using neural networks.

Neural networks are machine learning models that employ one or morelayers of nonlinear units to predict an output for a received input.Some neural networks include one or more hidden layers in addition to anoutput layer. The output of each hidden layer is used as input to thenext layer in the network, i.e., the next hidden layer or the outputlayer. Each layer of the network generates an output from a receivedinput in accordance with current values of a respective set ofparameters.

SUMMARY

This specification describes a capsule neural network implemented ascomputer programs on one or more computers in one or more locations.

According to a first aspect, there is provided a neural network that isconfigured to receive a network input and to generate a network outputfor the network input, wherein: the neural network comprises a pluralityof layers arranged in a sequence, including a plurality of capsulelayers; each capsule layer comprises a respective plurality of capsules;and each particular capsule in a particular capsule layer is configuredto: receive respective inputs including: (i) outputs generated bycapsules of a previous capsule layer that is before the particularcapsule layer in the sequence, and (ii) final routing factors betweencapsules of the previous capsule layer and the particular capsule,wherein the final routing factors are generated by a routing subsystem;and determine a particular capsule output based on the received inputs,wherein the particular capsule output is of dimension greater than one;and a routing subsystem that is configured to perform operationsincluding: receiving outputs generated by capsules in a first capsulelayer; determining final routing factors between capsules of the firstcapsule layer and capsules of a second capsule layer that is after thefirst capsule layer in the sequence; and providing the final routingfactors to the capsules in the second capsule layer.

In some implementations, determining final routing factors betweencapsules of the first capsule layer and capsules of a second capsulelayer that is after the first capsule layer in the sequence comprises:performing one or more iterations of an expectation-maximization routingalgorithm, wherein each iteration of the expectation-maximizationrouting algorithm includes an M-step and an E-step, wherein for aparticular iteration: the M-step includes, for each second capsule inthe second capsule layer: determining weighted votes for the secondcapsule from one or more first capsules of the first capsule layer forthe particular iteration, comprising: determining a vote for the secondcapsule from a particular first capsule of the first capsule layer bymultiplying an output of the particular first capsule by a weight matrixof the second capsule that is associated with the particular firstcapsule; weighting the vote for the second capsule from the particularfirst capsule of the first capsule layer by a routing factor between theparticular first capsule and the second capsule for the particulariteration; determining values of one or more parameters characterizing adistribution of weighted votes for the second capsule from the one ormore first capsules of the first capsule layer for the particulariteration; determining a cost associated with the second capsule for theparticular iteration based at least in part on the values of theparameters characterizing the distribution of weighted votes for thesecond capsule for the particular iteration; determining an activationof the second capsule for the particular iteration based at least inpart on the cost associated with the second capsule for the particulariteration; and the E, step includes determining routing factors for anext iteration based on at least the activations of the capsules of thesecond capsule layer for the particular iteration and the values of theparameters characterizing the distributions of weighted votes associatedwith the capsules of the second capsule layer for the particulariteration; determining final routing factors to be the routing factorsdetermined in a last iteration of the expectation-maximization routingalgorithm.

In some implementations, weighting the vote for the second capsule fromthe particular first capsule of the first capsule layer furthercomprises: weighting the vote for the second capsule from the particularfirst capsule by an activation of the particular first capsule.

In some implementations, the cost associated with the second capsule forthe particular iteration is further determined based at least in part onthe routing factors between the first capsules of the first capsulelayer and the second capsule for the particular iteration.

In some implementations, the cost associated with the second capsule forthe particular iteration is further determined based at least in part onan inverse temperature parameter.

In some implementations, the parameters characterizing the distributionof weighted votes for the second capsule for the particular iterationinclude: a mean of the weighted votes for the second capsule from theone or more first capsules of the first capsule layer for the particulariteration; and a variance of the weighted votes for the second capsulefrom the one or more first capsules of the first capsule layer for theparticular iteration.

In some implementations, determining final routing factors betweencapsules of the first capsule layer and capsules of a second capsulelayer that is after the first capsule layer in the sequence comprises:performing one or more iterations of a dynamic routing algorithm,wherein a particular iteration of the dynamic routing algorithmincludes: for each second capsule of the second capsule layer:determining weighted votes for the second capsule from one or more firstcapsules of the first capsule layer for the particular iteration,comprising: determining a vote for the second capsule from a particularfirst capsule of the first capsule layer by multiplying the output ofthe particular first capsule by a weight matrix of the second capsulethat is associated with the particular first capsule; weighting the votefor the second capsule from the particular first capsule of the firstcapsule layer by a normalized routing factor between the particularfirst capsule and the second capsule for the particular iteration;determining routing factors for a next iteration, wherein determining arouting factor between a first capsule of the first capsule layer and asecond capsule of the second capsule layer for the next iterationcomprises: determining a sum of the weighted votes for the secondcapsule from the one or more first capsules of the first capsule layerfor the particular iteration; determining a similarity measure betweenthe sum of the weighted votes for the second capsule for the particulariteration and the vote for the second capsule from the first capsule;determining the routing factor between the first capsule and the secondcapsule for the next iteration by combining the determined similaritymeasure with the routing factor between the first capsule and the secondcapsule for the particular iteration; determining final routing factorsto be the routing factors determined in a last iteration of the dynamicrouting algorithm.

In some implementations, determining the sum of the weighted votes forthe second capsule further comprises normalizing the sum of the weightedvotes for the second capsule.

In some implementations, the similarity measure between the sum of theweighted votes for the second capsule and the vote for the secondcapsule from the first capsule comprises: the inner product of the sumof the weighted votes for the second capsule and the vote for the secondcapsule from the first capsule.

In some implementations, determining a particular capsule output basedon the received inputs comprises: determining weighted votes for theparticular capsule from the capsules of the previous capsule layer,comprising: determining a vote for the particular capsule from a givencapsule in the previous capsule layer by multiplying the output of thegiven capsule by a weight matrix of the particular capsule that isassociated with the given capsule; weighting the vote for the particularcapsule from the given capsule in the previous capsule layer by a finalrouting factor between the given capsule and the particular capsule;determining the particular capsule output by combining the weightedvotes for the particular capsule from the capsules of the previouscapsule layer.

In some implementations, the final routing factors are normalized.

In some implementations, weighting the vote for the particular capsulefrom the given capsule in the previous capsule layer further comprises:weighting the vote for the particular capsule from the given capsule byan activation of the given first capsule.

In some implementations, combining the weighted votes for the particularcapsule from the capsules of the previous capsule layer comprises:determining a mean of the weighted votes for the particular capsule fromthe capsules of the previous capsule layer.

In some implementations, determining a particular capsule output basedon the received inputs further comprises: determining values of one ormore parameters characterizing a distribution of the weighted votes forthe particular capsule from the capsules of the previous capsule layer;determining a cost associated with the particular capsule based at leastin part on the values of the parameters characterizing the distributionof the weighted votes for the particular capsule; determining anactivation of the particular capsule based at least in part on the costassociated with the particular capsule.

In some implementations, the output of a capsule comprises an activityvector.

In some implementations, the output of a capsule comprises an activitymatrix.

In some implementations, the neural network comprises an initialconvolutional layer followed by a primary capsule layer.

In some implementations, the primary capsule layer is followed by aplurality of convolutional capsule layers.

In some implementations, a last convolutional capsule layer of theplurality of convolutional capsule layers is followed by a class capsulelayer.

In some implementations, for each capsule of the class capsule layer,weight matrices of the capsule that are associated with capsules of thelast convolutional capsule layer of a same type share weight values,wherein: a first capsule of a convolutional capsule layer is of a sametype as a second capsule of the convolutional capsule layer if weightmatrices of the first capsule share weight values with weight matricesof the second capsule; values based on the coordinates of the center ofthe receptive fields of capsules of the last convolutional capsule layerare added to votes of capsules of the last convolutional capsule layer.

In some implementations, the neural network is configured to receive anetwork input and to classify the network input as belonging to one ormore of a predetermined set of classes, and wherein the class capsulelayer includes a respective capsule for each class in the predeterminedset.

In some implementations, an activation of a respective capsule of theclass capsule layer is greater than activations of other capsules of theclass capsule layer; the network input is classified as belonging to aparticular class corresponding to the respective capsule of the classcapsule layer.

In some implementations, the network input is an image,

The above aspects can be implemented in any convenient form, Forexample, aspects and implementations may be implemented by appropriatecomputer programs which may be carried on appropriate carrier mediawhich may be tangible carrier media (e.g. disks) or intangible carriermedia (e.g. communications signals). Aspects may also be implementedusing suitable apparatus which may take the form of programmablecomputers running computer programs.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages, Capsule neural networks can include a dynamicrouting system to route the outputs of lower layers of the capsulenetwork to higher layers of the capsule network, where the routing ofoutputs between capsule network layers is dynamically determined basedon the outputs themselves. The dynamic routing system described in thisspecification facilitates superior performance (e.g., superior neuralnetwork prediction accuracy) relative to conventional routing methods.In some conventional routing methods, the routing of outputs betweenneural network layers is pre-determined, and in some other conventionalrouting methods such as max-pooling methods, the routing is dynamicallydetermined by routing only the largest outputs and discarding theothers. The dynamic routing system as described in this specificationfacilitates superior performance relative to such conventional routingmethods because it enables a more flexible routing of outputs, therebyallowing a neural network including the dynamic routing system asdescribed in this specification to route relevant information moreeffectively and generate more accurate predictions as a result.

Capsule networks as described in this specification may consume fewercomputational resources than conventional neural networks. For example,in the setting of image processing and computer vision, a conventionalneural network that is configured to process input images is typicallytrained on a large number of input images depicting objects and scenesfrom different viewpoints. Training a conventional neural network inthis manner may endow the conventional neural network with some capacityto make accurate predictions about objects and scenes depicted from avariety of viewpoints. Capsule networks as described in thisspecification can naturally generalize to make accurate predictionsabout objects and scenes depicted from a variety of viewpoints, evenwhen trained on less training data than conventional neural networks.Since training a neural network based on a larger training set consumesmore computational resources, training capsule networks may consumefewer computational resources (e.g., memory and processing power) thantraining conventional neural networks. Furthermore, since capsulenetworks as described in this specification can naturally generalize tomake accurate predictions about objects and scenes depicted from avariety of viewpoints, a capsule network that has similar predictionaccuracy as a conventional neural network can (in some situations) beimplemented with fewer parameters than the conventional neural network,thereby reducing the amount of memory required to maintain the neuralnetwork, the amount of data. required to train the neural network, andthe amount of processing power required to process inputs using theneural network.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an illustration of an example prediction system.

FIG. 1B is an illustration of an example architecture of a capsuleneural network.

FIG. 2 is a flow diagram of an example process for determining finalrouting factors between capsule layers.

FIG. 3A is a flow diagram of an example process for determining finalrouting factors between a first capsule layer and a second capsule layer(that is after the first capsule layer) using anexpectation-maximization routing algorithm.

FIG. 3B is an illustration of an example data flow of an iteration of amaximization step of an expectation maximization routing algorithm.

FIG. 4 is a flow diagram of an example process for determining finalrouting factors between a first capsule layer and a second capsule layer(that is after the first capsule layer) using a dynamic routingalgorithm.

FIG. 5 is an example process for determining a capsule output and acapsule activation.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

This specification generally describes capsule neural networks that areimplemented by one or more computers in one or more locations. A capsuleneural network includes multiple layers arranged in a sequence,including one or more capsule layers, and in some cases, one or moreneuron layers.

A neuron layer refers to a conventional neural network layer composed ofneurons. Each neuron is configured to generate a neuron output. A neuronoutput is a single number. Each neuron has a set of parameter values andan activation function. A neuron generates a neuron output by computinga weighted linear combination of neuron inputs and applying anactivation function to the weighted linear combination. The weights ofthe linear combination are given by the parameter values of the neuron.Generally, activation functions are a non-linear function of theweighted linear combination. Examples of activation functions includerectified linear unit activation functions and sigmoid activationfunctions.

A convolutional neuron layer is a neuron layer where the neurons (andtheir outputs) may be logically organized into a rectangular array witha length, a width, and a depth. Each of the neurons corresponding to aparticular depth is referred to as being of a particular neuron type.Neurons of the same type within a given convolutional neuron layer shareparameter values.

A capsule layer refers to a neural network layer composed of capsules.Each capsule is configured to generate a capsule output and, in somecases, a capsule activation. The capsule output includes two or morenumbers and may be represented as a vector or as a matrix. A capsulegenerates a capsule output by processing a capsule input in accordancewith a set of capsule parameters. In general, the capsule activation isa number (i.e., not a vector or a matrix) An example process fordetermining a capsule output and a capsule activation is described withreference to FIG. 5.

In some cases, capsules in a given capsule layer are configured toreceive as input capsule outputs of one or more capsules of a previouscapsule layer and final routing factors that are determined by a routingsystem. In these cases, the set of capsule parameters includes one ormore weight matrices, where each weight matrix of a capsule correspondsto a capsule of a previous capsule layer.

In some other cases, capsules in a given capsule layer are configured toreceive as input neuron outputs of neurons of a previous neuron layer.Such capsules are not configured to receive final routing factors thatare determined by the routing system.

The receptive field of a particular capsule refers to the neurons orcapsules of previous layers which have outputs that are provided asinput, either directly or indirectly, to the particular capsule. Anoutput of a neuron or capsule is said to be indirectly provided as inputto a particular capsule if the output of the neuron or capsule affectsthe output generated by the particular capsule. For example, if a firstcapsule provides an input directly to a second capsule, and the secondcapsule provides an input directly to a third capsule, then the outputof the first capsule is said to be indirectly provided as input to thethird capsule.

A convolutional capsule layer is a capsule layer where the capsules arelogically organized into a rectangular array with a length, a width, anda depth. Each of the capsules corresponding to a particular depth isreferred to as being of a particular capsule type. Capsules of the sametype within a given convolutional capsule layer share parameter values.

In this specification, a primary capsule layer refers to a capsule layerthat is configured to receive as input neuron outputs of neurons of aprevious neuron layer. In some cases, primary capsule layers may beconvolutional capsule layers.

The routing system determines how to route outputs between capsulelayers of the capsule network. Specifically, the routing system receivesoutputs generated by capsules of one capsule layer of the capsulenetwork and determines how to route the outputs to capsules of anothercapsule layer that is after the capsule layer in the capsule network.The routing system determines how to route the outputs by determiningfinal routing factors between the capsules of the first capsule layerand the capsules of the second capsule layer. Routing factors are scalarvalues that are associated with pairs of capsules of the capsulenetwork.

Capsule networks can be trained to perform prediction tasks. In general,capsule networks can be used in place of conventional convolutionalneural networks (i.e., networks with multiple convolutional neuronlayers but without capsule layers) or other conventional neuralnetworks.

For example, a capsule network can be configured to receive as input animage and to process the image to perform image classification or objectdetection tasks. For image classification tasks, the capsule network maygenerate outputs indicating respective probabilities that input imagesdepict different types of objects (e.g., dogs or cats). For objectdetection tasks, the capsule network may generate outputs indicatinglocations of different types of objects depicted in input images (e.g.,coordinates of bounding boxes around the different types of objects inthe input images).

Capsule networks can be configured to perform other tasks, such as videoanalysis. For video analysis, the capsule network is configured toreceive as input video data and to process the video data to performvideo classification (e.g., to predict the topic of the video). Capsulenetworks can also be configured to perform natural language processingtasks, where the capsule network is configured to receive as input textdata and to process the text data to perform, for example, machinetranslation. Capsule networks can also be configured to perform speechrecognition, where the capsule network is configured to receive as inputaudio data and to process the audio data to generate as output a texttranscription of a spoken utterance from the audio data.

FIG. 1A is an illustration of an example prediction system. Theprediction system 128 is an example of a system implemented as computerprograms on one or more computers in one or more locations in which thesystems, components, and techniques described below are implemented.

The prediction system 128 includes a capsule neural network 100. Thearchitecture of the capsule neural network 100 described in thisspecification is an example of a possible capsule network architecture.In general, different capsule network architectures may be implemented.

The network 100 is configured to receive a network input 102 and togenerate a network output 118 for the network input 102. In someimplementations, the network 100 is configured to classify the networkinput 102 as belonging to one or more of a predetermined set of classes.For example, the network 100 may be configured to perform imageclassification tasks, where the network input 102 is an image and thenetwork output 118 indicates a class of object depicted in the inputimage.

The network 100 provides the network input 102 to a convolutional neuronlayer 104 that is configured to process the network input 102 inaccordance with a set of convolutional neuron layer parameters togenerate a convolutional neuron layer output 106. As previouslydescribed, the convolutional neuron layer output 106 may be logicallyorganized into a rectangular array with a length, a width, and a depth.

The network 100 provides the convolutional neuron layer output 106 asinput to a primary capsule layer 108 that is configured to process theconvolutional neuron layer output 106 in accordance with a set ofprimary capsule layer parameters to generate a primary capsule layeroutput 110. In some implementations, the primary capsule layer 108 is aconvolutional capsule layer. In some implementations, for each capsuleof the primary capsule layer 108, the capsule determines each componentof the capsule output (i.e., each element of the capsule outputvector/matrix) as a different linear combination of the components ofthe convolutional neuron layer output 106 (i.e., the elements of therectangular array of convolutional neuron layer outputs). In someimplementations, for each capsule of the primary capsule layer 108, thecapsule determines a capsule activation by applying an activationfunction (e.g., a rectified linear unit activation function) to a linearcombination of the components of the convolutional neuron layer output106. The weights of the linear combinations that the capsule uses todetermine the capsule output and the capsule activation are parametersof the capsule. In some implementations, the receptive field of eachcapsule of the primary capsule layer in the convolutional neuron layer104 is a respective column of the convolutional neuron layer 104. Acolumn refers to a set of neurons of the convolutional neuron layer 104that have the same length and width position but different depthpositions.

The network 100 provides the primary capsule layer output 110 to arouting system 120. The routing system 120 processes the primary capsulelayer output 110 to generate as output final routing factors 122. Eachof the final routing factors 122 is a scalar value that is associatedwith a pair of capsules including (i) a capsule of the convolutionalcapsule layer 112 and (ii) a capsule of the primary capsule layer 108that is in the receptive field of the capsule of the convolutionalcapsule layer 112. An example process for determining final routingfactors between capsule layers is described with reference to FIG. 2.

The network provides the final routing factors 122 and the primarycapsule layer output 110 as input to the convolutional capsule layer112. For each particular capsule of the convolutional capsule layer 112,the particular capsule processes the outputs of capsules in the primarycapsule layer 108 that are in the receptive field of the particularcapsule. The particular capsule also processes the final routing factors122 that are associated with pairs of capsules including both theparticular capsule and a capsule in the primary capsule layer 108 in thereceptive field of the particular capsule. The particular capsuleprocesses the respective outputs and routing factors in accordance witha set of parameters including one or more weight matrices, where eachweight matrix of the particular capsule corresponds to a capsule of theprimary capsule layer 108 in the receptive field of the particularcapsule. The particular capsule generates an output (which is in generala vector or a matrix), and in some cases, an activation (which is ingeneral a scalar value). An example process for determining a capsuleoutput and a capsule activation is described with reference to FIG. 5.The convolutional capsule layer output 114 is composed of the outputsgenerated by the capsules of the convolutional capsule layer 112.

In some implementations, the network 100 includes multiple additionalcapsule layers after the convolutional capsule layer 112 and before theclass capsule layer 116. For each additional capsule layer, theadditional capsule layer is configured to process outputs generated by aprevious capsule layer and final routing factors between the previouscapsule layer and the additional capsule layer generated by the routingsystem 120, to generate a respective capsule layer output.

The class capsule layer 116 is the last capsule layer of the network100. In some implementations, the network 100 is configured to classifythe network input 102 as belonging to one or more of a predetermined setof classes. In some of these implementations, the class capsule layer116 includes a respective capsule for each class in the predeterminedset of classes. In some of these implementations, each class of thepredetermined set of classes may correspond to a class of object. Theclass capsule layer 116 is configured receive outputs generated by aprevious capsule layer and routing factors generated by the routingsystem 120 between the previous capsule layer and the class capsulelayer, and to process these inputs to generate a network output 118.

In some implementations, the network output 118 is the activation ofeach capsule of the class capsule layer 116. In some of theseimplementations, the network input 102 is determined to belong to aparticular class in the predetermined set of classes if the activationof the capsule of the class capsule layer 116 corresponding to theparticular class is greater than the activations of other capsules ofthe class capsule layer 116.

In some other implementations, the network output 118 is the output ofeach capsule of the class capsule layer 116. In some of theseimplementations, the network input 102 is determined to belong to aparticular class in the predetermined set of classes if the norm (e.g.,the Euclidean norm) of the output of the capsule of the class capsulelayer 116 corresponding to the particular class is greater than thenorms of the outputs of other capsules of the class capsule layer 116.

In some implementations, the system 128 includes a training engine 126that adjusts the parameters of the network 100 based on a loss function124. In some implementations, the network 100 is configured to classifythe network input 102 as belonging to one or more of a predetermined setof classes, the class capsule layer 116 includes a respective capsulefor each class in the predetermined set, and the network output 118 isthe output of each capsule of the class capsule layer 116. In some ofthese implementations, the loss function 124 is given by a sum of lossterms, where each loss term corresponds to a different class capsule,and the loss term corresponding to class capsule c is given by:

T _(c)max(0, m ⁺ −|v _(c)|)²+λ(1−T _(c))max(0, |v _(c) |−m ⁻)²,

where T_(c)=1 if and only if the network input 102 belongs to class c,|v_(c)| is norm of the output of class capsule c (e.g., the Euclideannorm), and m⁺, m⁻, and λ are fixed hyperparameters of the loss functionthat may be selected in any appropriate manner (e.g., arbitrarily or bya cross-validation process). In some implementations, the network output118 is the activation of each capsule of the class capsule layer 116,and the loss function 124 is given by:

${\sum\limits_{i \neq t}L_{i}},$

where:

L _(i)=(max(0, m−(a _(t) −a _(i)))²,

where the network input 102 belongs to class t, a_(t) is the activationof class capsule t, a_(i) is the activation of class capsule i, m is afixed hyperparameter of the loss function that may be selected in anyappropriate manner, and the sum is over each class of the predeterminedset of classes.

The training engine 126 adjusts the parameters of the network 100 basedon the loss function 124 by determining network outputs 118 for inputsin a set of training data. For each input in the set of training data,the class membership of the input is known. For example, if the trainingengine 126 trains the network 100 to perform image classification tasks,then the training data may include images and corresponding image classlabels, where the image class label indicates the type of objectdepicted in the corresponding image. Based on the network outputsgenerated by the network 100 for the inputs of the training data, thetraining engine 126 backpropagates gradients of the loss function 124 toadjust the parameters of the network 100. The training engine 126 canuse any appropriate neural network training technique, such asstochastic gradient descent. The parameters of the network 100 that areadjusted during the training include parameters of the neuron layers(e.g., the convolutional neuron layer 104), the capsule layers (e.g.,the primary capsule layer 108, the convolutional capsule layer 112, andthe class capsule layer 118), and the routing system 120.

In this specification, parameters referred to as vectors or matrices canbe stored by the network 100 in any appropriate format that enablestheir use in the manner described in this specification. For example,parameters referred to as matrices need not be stored by the network 100in the form of matrices (e.g., they may be stored as vectors or in anyother appropriate format).

FIG. 1B is an illustration of an example architecture of a capsuleneural network, e.g., the capsule neural network 100 of FIG. 1A. Thecapsule neural network 130 receives as input an image 132, and processesthe input with a convolutional neuron layer 134, a primary capsule layer136, convolutional capsule layers 138 and 140, and a class capsule layer142. The receptive fields of the neurons and capsules of the respectivelayers are illustrated. For example, the neuron 144 of the convolutionalneuron layer 134 has receptive field 146 in the image 132.

FIG. 2 is a flow diagram of an example process for determining finalrouting factors between capsule layers. For convenience, the process 200will be described as being performed by a system of one or morecomputers located in one or more locations. For example, a routingsystem, e.g., the routing system 120 of FIG. 1A, appropriatelyprogrammed in accordance with this specification, can perform theprocess 200.

The system receives outputs generated by a first capsule layer of thecapsule network (202). For example, the system may receive outputsgenerated by a convolutional capsule layer, or any other kind of capsulelayer.

The system determines a routing of the outputs to a second capsule layerof the capsule network that is after the first capsule layer (202).Determining a routing of the outputs includes determining final routingfactors between the capsules of the first capsule layer and capsules ofthe second capsule layer. An example process for determining finalrouting factors using an expectation-maximization routing algorithm isdescribed below with reference to FIG. 3A. An example process fordetermining final routing factors using a dynamic routing algorithm isdescribed below with reference to FIG. 4.

The system provides the final routing factors to the second capsulelayer of the capsule network (206).

FIG. 3A is a flow diagram of an example process for determining finalrouting factors between a first capsule layer and a second capsule layer(that is after the first capsule layer) using anexpectation-maximization routing algorithm. For convenience, the process300 will be described as being performed by a system of one or morecomputers located in one or more locations. For example, a routingsystem, e.g., the routing system 120 of FIG. 1A, appropriatelyprogrammed in accordance with this specification, can perform theprocess 300.

The expectation-maximization routing algorithm is an iterativealgorithm. Each iteration of the expectation-maximization routingalgorithm includes two steps: a maximization step, referred to as anM-step (302), that includes 306, 308, 310, and 312, and an expectationstep, referred to as an E-step (304), that includes 314. An example dataflow of an iteration of a maximization step of an expectationmaximization routing algorithm is described with reference to FIG. 3B.

For each second capsule of the second capsule layer, the systemdetermines weighted votes for the second capsule from the first capsulesof the first capsule layer in the receptive field of the second capsule(306).

In particular, for a given second capsule in the second capsule layer,the system determines a respective vote for the given second capsulefrom each particular first capsule of the first capsule layer in thereceptive field of the second capsule. Each second capsule of the secondcapsule layer has a respective weight matrix corresponding to eachparticular first capsule of the first capsule layer in the receptivefield of the second capsule. To determine a vote for the second capsulefrom a particular first capsule, the system multiplies the output of theparticular first capsule by the weight matrix of the second capsule thatcorresponds to the particular first capsule. In some implementations,the output of the particular first capsule is a vector. In theseimplementations, the weight matrix of the second capsule correspondingto the particular first capsule is a matrix with a number of columnsthat is the same as the dimensionality of the output of the particularfirst capsule and at least two rows. For example, if the output of theparticular first capsule is a n×1 vector, then the weight matrix of thesecond capsule corresponding to the particular first capsule is a r×nmatrix, where r is an integer value of two or greater. In someimplementations, the output of the particular first capsule is a matrix.In these implementations, the weight matrix of the second capsulecorresponding to the particular first capsule is a matrix with at leasttwo rows and with a number of columns that is the same as the number ofrows of the output of particular first capsule. For example, if theoutput of the particular first capsule is a n×n matrix, then the weightmatrix of the second capsule corresponding to the particular firstcapsule is a r×n matrix, where r is an integer vale of two or greater.

In some implementations, when the first capsule layer and the secondcapsule layer are both convolutional capsule layers, the weight matricesof capsules of a second type in the second capsule layer correspondingto capsules of a first type in the first capsule layer are all shared(i.e., the weight matrices are the same). In some of theseimplementations, the system adds the coordinates of the center of thereceptive field of the particular first capsule in the input image tothe vote for the second capsule from the particular first capsule. Insome implementations, the coordinates of the center of the receptivefield of the first capsule are given by the x-y coordinates (in theframe of reference of the input image) of a pixel that is determined tobe the center (e.g., the centroid) of the set of pixels of the inputimage in the receptive field of the first capsule. In someimplementations, the coordinates of the center of the receptive field ofthe particular first capsule may be scaled (e.g., such that eachcoordinate is in the range 0 to 1). In some implementations, when thevote for the second capsule from the particular first capsule is amatrix, the system adds the coordinates of the center of the receptivefield of the particular first capsule to the first two components of therightmost column of the vote for the second capsule from the particularfirst capsule (e.g., the x-coordinate of the center of the receptivefield may be added to the first component of the rightmost column of thevote and the y-coordinate may be added to the second component of therightmost column of the vote).

For each particular first capsule of the first capsule layer in thereceptive field of the second capsule, the system weights the vote forthe second capsule from the particular first capsule by the value of arouting factor between the particular first capsule and the secondcapsule for the current iteration. Weighting a vote refers tomultiplying the vote by a scalar value. For iterations after the firstiteration of the expectation-maximization routing algorithm, the valueof the routing factor for the current iteration is determined in aprevious iteration of the expectation-maximization algorithm. For thefirst iteration of the expectation-maximization algorithm, the value ofthe routing factor may be given by an arbitrary value. For example, forthe first iteration of the expectation-maximization routing algorithm,the values of the routing factors between capsules of the first capsulelayer and capsules of the second capsule layer may be initialized to auniform value. In some implementations, the system additionally weightsthe vote for the second capsule from the particular first capsule by thevalue of an activation of the particular first capsule.

For each second capsule in the second capsule layer, the systemdetermines values of parameters characterizing a distribution ofweighted votes for the second capsule from the one or more firstcapsules of the first capsule layer that are in the receptive field ofthe second capsule (308).

In sonic implementations, the parameters characterizing the distributionof weighted votes for the second capsule from the one or more firstcapsules of the first capsule layer includes a mean of the weightedvotes. For example, the parameters characterizing the distribution ofweighted votes for the second capsule may include a parameter given by:

${\mu = \frac{\Sigma_{i}r_{i}a_{i}V_{i}}{\Sigma_{i}r_{i}a_{i}}},$

where r_(i) is the routing factor between capsule i of the first capsulelayer and the second capsule for the current iteration, a_(i) is anactivation of capsule i, and V_(i) is the vote of capsule i for thesecond capsule. In this case, the distribution parameter μ has the samedimensionality (i.e., the same vector or matrix dimensionality) as thevotes {V_(i)} of the capsules of the first capsule layer.

In some implementations, the parameters characterizing the distributionof weighted votes for the second capsule from the one or more firstcapsules of the first capsule layer includes a variance of the weightedvotes. For example, the parameters characterizing the distribution ofweighted votes for the second capsule may include parameters {σ_(h)²}_(h), where h is an index over the components of votes for secondcapsules of the second capsule layer from first capsules of the firstcapsule layer, where:

${\sigma_{h}^{2} = \frac{\Sigma_{i}r_{i}{a_{i}\left( {V_{ih} - \mu_{h}} \right)}^{2}}{\Sigma_{i}r_{i}a_{i}}},$

where r_(i) is the routing factor between capsule i of the first capsulelayer and the second capsule for the current iteration, a_(i) is anactivation of capsule i, and V_(ih) is the h-th component of the vote ofcapsule i for the second capsule, and μ_(h) is the h-th component of themean of the weighted votes (as previously described).

For each second capsule in the second capsule layer, the systemdetermines a cost associated with the second capsule based on at leastthe values of the parameters characterizing the distribution of weightedvotes associated with the second capsule (310). In some implementations,the cost associated with the second capsule is further determined basedon at least the routing factors between: (i) the one or more firstcapsules in the receptive field of the second capsule and (ii) thesecond capsule (for the current iteration). For example, the costassociated with the second capsule may be given by {cost_(h)}_(h), whereh is an index over the components of votes for second capsules of thesecond capsule layer from first capsules of the first capsule layer,where:

${{cost}_{h} = {\left( {\beta_{bv} + {\log \left( \sigma_{h} \right)}} \right){\sum\limits_{i}{r_{i}a_{i}}}}},$

where the sum is over all capsules i in the first capsule layer that arein the receptive field of the second capsule, r_(i) is the routingfactor between capsule i of the first capsule layer and the secondcapsule for the current iteration, a_(i) is an activation of capsule i,σ_(h) ² is the h-th component of the variance of the weighted votes (aspreviously described), and β_(v) is a parameter of the cost function. Insome cases, the system determines the value of the parameter β_(v)during the training of the capsule network (e.g., by iterativelyadjusting the value of the parameter β_(v) by backpropagatinggradients).

For each second capsule in the second capsule layer, the systemdetermines an activation of the second capsule (312). In someimplementations, the system determines the activation of the secondcapsule based on the cost associated with the second capsule (i.e., thecost computed in 310). For example, the system may determine theactivation of the second capsule as:

${a = {{sigmoid}\left( {\lambda\left( {\beta_{a} - {\sum\limits_{h}{cost}_{h}}} \right)} \right)}},$

where the sum is over the components h of the votes for the secondcapsule, sigmoid refers to the sigmoid function, λ is an inversetemperature parameter, β_(a) is a parameter of the activation function,and cost_(h) is the h-th component of the cost associated with thesecond capsule. In some cases, the system determines the value of theparameter β_(a) during training of the capsule network (e.g., byiteratively adjusting the value of the parameter β_(a) bybackpropagating gradients). In some cases, the value of the inversetemperature parameter λ changes between iterations of theexpectation-maximization routing algorithm in accordance with a fixedschedule. More specifically, in some cases, the value of the inversetemperature parameter λ increases between iterations of theexpectation-maximization routing algorithm in accordance with a fixedschedule. For example, the value of the inverse temperature parametermay increase by a fixed factor at each iteration.

The system determines values of routing factors for the next iterationof the expectation-maximization routing algorithm (314). In someimplementations, the system determines the values of the routing factorsfor the next iteration based on at least the activations of the capsulesof the second capsule layer for the current iteration and the values ofthe parameters characterizing the distributions of weighted votesassociated with the capsules of the second capsule layer for the currentiteration. For example, for each particular capsule of the first capsulelayer, the system may determine the values of the routing factorsbetween the particular capsule and capsules in the second capsule layerthat receive votes from the particular capsule for the next iterationas:

${p_{c} = {\frac{1}{\sqrt{\Pi_{h}2{\pi\sigma}_{ch}^{2}}}{\exp\left( {- {\sum\limits_{h}\frac{\left( {V_{ch} - \mu_{ch}} \right)^{2}}{2\sigma_{ch}^{2}}}} \right)}}},{r_{c} = \frac{a_{c}p_{c}}{\Sigma_{j}a_{j}p_{j}}},$

where c and j index the second capsules of the second capsule layer thatreceive votes from the particular capsule of the first capsule layer, hindexes the components of the votes for capsules of the second layerfrom capsules of the first layer, V_(ch), is the h-th component of thevote of the particular capsule of the first capsule layer and capsule cof the second capsule layer, σ_(ch) ², is the h-th component of thevariance of the weighted votes for capsule c (as previously described),μ_(ch) is the h-th component of the mean of the weighted votes forcapsule c (as previously described), r_(c) is the routing factor for thenext iteration between the particular capsule and the capsule c of thesecond capsule layer, and a_(c) is the activation of the second capsulec.

The system determines whether the current iteration is the finaliteration of the expectation-maximization routing algorithm (316). Insome implementations, the system may determine that the currentiteration is the final iteration if the current iteration is the lastiteration of a pre-determined number of iterations. In someimplementations, the system may determine that the current iteration isthe final iteration if the values of the routing factors betweencapsules of the first capsule layer and capsules of the second capsulelayer determined for the current iteration are sufficiently similar tothe values of the routing factors determined for the previous iteration(e.g., have an average difference that is less than a predeterminedthreshold). In some implementations, the system may determine that thecurrent iteration is the final iteration if the activations of thecapsules of the second capsule layer determined for the currentiteration are sufficiently similar to the activations of the capsules ofthe second capsule layer determined for the previous iteration.

In response to determining that the current iteration is not the finaliteration, the system returns to 306 and repeats the preceding steps.

In response to determining that the current iteration is the finaliteration, the system determines the final routing factors (318). Insome implementations, the system determines the final routing factors tobe the routing factors determined in the final iteration of theexpectation-maximization routing algorithm.

An example data flow of an iteration of a maximization step of anexpectation-maximization routing algorithm for a second capsule of asecond capsule layer that is after a first capsule layer is describedwith reference to FIG. 3B.

Each capsule of the first capsule layer (e.g., 324, 326, . . . ) that isin the receptive field of the second capsule 322 has a capsule output(e.g., 328, 330, . . . ). The system multiples the capsule outputs ofthe first capsule layer by weight matrices (e.g., 332, 334, . . . ) todetermine votes (e.g., 336, 338, . . . ). The system weights the votesby weighting factors (e.g., 340, 342, . . . ) to determine weightedvotes (e.g., 344, 346, . . . ). 348 illustrates the weighted votes forthe second capsule in a 2-dimensional space. In general, the weightedvotes for the second capsule may belong to a space with a dimension ofhigher than 2. The system determines distribution parameters 352 thatcharacterize the distribution of weighted votes for the second capsule.The system determines a cost 352 and an activation 354 of the secondcapsule.

FIG. 4 is a flow diagram of an example process for determining finalrouting factors between a first capsule layer and a second capsule layer(that is after the first capsule layer) using a dynamic routingalgorithm. For convenience, the process 400 will be described as beingperformed by a system of one or more computers located in one or morelocations. For example, a routing system, e.g., the routing system 120of FIG. 1A, appropriately programmed in accordance with thisspecification, can perform the process 400.

The system determines normalized values of routing factors betweencapsules of the first capsule layer and capsules of the second capsulelayer for the current iteration (401).

For iterations after the first iteration of the dynamic routingalgorithm, the values of the routing factors for the iteration aredetermined in a previous iteration of the dynamic routing algorithm. Insome implementations, the system determines the routing factors for thefirst iteration of the dynamic routing algorithm to be arbitrary values.For example, the system may initialize the values of the routing factorsfor the first iteration to a uniform value. In some implementations, thesystem learns the values of the routing factors for the first iterationduring training (e.g., by iteratively adjusting the values of therouting factors for the first iteration by backpropagating gradients).In these implementations, the value of a routing factor between a firstcapsule of the first capsule layer and a second capsule of the secondcapsule layer for the first iteration of the dynamic routing algorithmdepends on the type and location of the first and second capsules, butnot the network input.

In some implementations, the system determines normalized values ofrouting factors between capsules of the first capsule layer and capsulesof the second capsule layer for the current iteration as:

c _(i)=softmax(b _(i)),

where the system performs the normalization for each first capsule i ofthe first capsule layer, b_(i) is a vector of routing factors betweenfirst capsule i and second capsules of the second capsule layer thatreceive votes from capsule i for the current iteration, and c_(i) is avector of normalized values of the routing factors between first capsulei and second capsules of the second capsule layer that receive votesfrom capsule i for the current iteration.

For each second capsule in the second capsule layer, the systemdetermines a sum of weighted votes for the second capsule from the firstcapsules of the first capsule layer in the receptive field of the secondcapsule (402).

In particular, for a given second capsule in the second capsule layer,the system determines a respective vote for the given second capsulefrom each particular first capsule of the first capsule layer in thereceptive field of the second capsule. Each second capsule of the secondcapsule layer has a respective weight matrix corresponding to eachparticular first capsule of the first capsule layer in the receptivefield of the second capsule. To determine a vote for the second capsulefrom a particular first capsule, the system multiplies the output of theparticular first capsule by the weight matrix of the second capsule thatcorresponds to the particular first capsule. In some implementations,the output of the particular first capsule is a vector. In theseimplementations, the weight matrix of the second capsule correspondingto the particular first capsule is a matrix with at least two rows and anumber of columns that is the same as the dimensionality of the outputof the particular first capsule. For example, if the output of theparticular first capsule is a n×1 vector, then the weight matrix of thesecond capsule corresponding to the particular first capsule is a r×nmatrix, where r is an integer value of at least two. In someimplementations, the output of the particular first capsule is a matrix.In these implementations, the weight matrix of the second capsulecorresponding to the particular first capsule is a matrix that has atleast two rows and a number of columns that is the same as the number ofrows of the output of the particular first capsule. For example, if theoutput of the particular first capsule is a n×n matrix, then the weightmatrix of the second capsule corresponding to the particular firstcapsule is a r×n matrix, where r is an integer value of at least two.

In some implementations, when the first capsule layer and the secondcapsule layer are both convolutional capsule layers, the weight matricesof capsules of a second type in the second capsule layer correspondingto capsules of a first type in the first capsule layer are all shared(i.e., the weight matrices are the same). In some of theseimplementations, the system adds the coordinates of the center of thereceptive field of the particular first capsule in the input image tothe vote for the second capsule from the particular first capsule. Insome implementations, the coordinates of the center of the receptivefield of the first capsule are given by the x-y coordinates (in theframe of reference of the input image) of a pixel that is determined tobe the center (e.g., the centroid) of the set of pixels of the inputimage that are in the receptive field of the first capsule. In someimplementations, the coordinates of the center of the receptive field ofthe particular first capsule may be scaled (e.g., such that eachcoordinate is in the range 0 to 1). In some implementations, when thevote for the second capsule from the particular first capsule is amatrix, the system adds the coordinates of the center of the receptivefield of the particular first capsule to the first two components of therightmost column of the vote for the second capsule from the particularfirst capsule (e.g., the x-coordinate of the center of the receptivefield may be added to the first component of the rightmost column of thevote and the y-coordinate may be added to the second component of therightmost column of the vote).

In some implementations, the system determines the sum of weighted votesfor each second capsule j of the second capsule layer as:

${s_{j} = {\sum\limits_{i}{c_{ij}V_{ji}}}},$

where i indexes capsules of the first capsule layer that are in thereceptive field of capsule j, c_(ij) is the normalized routing factorbetween capsule i and capsule j for the current iteration, and is thevote of capsule i of the first capsule layer for capsule j of the secondcapsule layer.

In some implementations, the system normalizes the sum of weighted votesfor each second capsule of the second capsule layer (404). For example,the system may normalize the sum of the weighted votes for each secondcapsule of the second capsule layer to have a norm (e.g., a Euclideannorm) between 0 and 1., e.g., by normalizing the sum of the weightedvotes for second capsule j of the second capsule layer as:

${v_{j} = {\frac{{s_{j}}^{2}}{1 + {s_{j}}^{2}}\frac{s_{j}}{s_{j}}}},$

where |s_(j)| is a norm (e.g., a Euclidean norm) of the sum of theweighted votes for capsule j.

The system determines routing factors for the next iteration of thedynamic routing algorithm (406). For each second capsule of the secondcapsule layer and each first capsule of the first capsule layer in thereceptive field of the second capsule, the system determines asimilarity measure between the sum of the weighted votes for the secondcapsule and the vote for the second capsule from the first capsule. Insome implementations, the system determines the similarity measurebetween the sum of the weighted votes for the second capsules and thevote for the second capsule from the first capsule by computing theinner product between them.

Next, the system determines the routing factor between the first capsuleand the second capsule for the next iteration by combining thedetermined similarity measure with the routing factor between the firstcapsule and the second capsule for the current iteration. For example,the system may determine the routing factor for the next iterationbetween the first capsule i of the first capsule layer and the secondcapsule j of the second capsule layer as:

r _(ij) ^(t+1) =r _(ij) ^(t) +V _(j|i) ·v _(j),

where r_(ij) ^(t+1) is the routing factor between capsule i of the firstcapsule layer and capsule j of the second capsule layer for the nextiteration, r_(ij) ^(t) is the routing factor between the capsules forthe current iteration, v_(j) is the normalized sum of weighted votes forsecond capsule j (as described previously), · indicates the innerproduct operation, and V_(j|i) is the vote of capsule i of the firstcapsule layer for capsule j of the second capsule layer.

The system determines whether the current iteration is the finaliteration of the dynamic routing algorithm (408). In someimplementations, the system may determine that the current iteration isthe final iteration if the current iteration is the last iteration of apre-determined number of iterations. In some implementations, the systemmay determine that the current iteration is the final iteration if thevalues of the routing factors between capsules of the first capsulelayer and capsules of the second capsule layer determined for thecurrent iteration are sufficiently similar to the values of the routingfactors determined for the previous iteration (e.g., have an averagedifference that is less than a predetermined threshold).

In response to determining that the current iteration is not the finaliteration, the system returns to 401 and repeats the preceding steps. Inresponse to determining that the current iteration is the finaliteration, the system determines values of final routing factors (410).In some implementations, the system determines the values of finalrouting factors to be the normalized values of the routing factorsdetermined in the final iteration of the dynamic routing algorithm(where the routing factor normalization can be performed as describedabove).

FIG. 5 is a flow diagram of an example process for determining a capsuleoutput and a capsule activation. Specifically, FIG. 5 describes anexample process for a second capsule of a second capsule layer thatreceives outputs of a first capsule layer (that is before the secondcapsule layer) to determine a second capsule output and a second capsuleactivation. For convenience, the process 500 will be described as beingperformed by a system of one or more computers located in one or morelocations. For example, a capsule of a capsule neural network, e.g., thecapsule neural network 100 of FIG. 1A, appropriately programmed inaccordance with this specification, can perform the process 500.

The second capsule receives final routing factors from the routingsystem and outputs of capsules in a first capsule layer that is beforethe second capsule layer (501). The second capsule receives finalrouting factors between first capsules of the first capsule layer (inthe receptive field of the second capsule) and the second capsule. Anexample process for a routing system to determine final routing factorsbetween a first capsule layer and a second capsule layer using anexpectation-maximization routing algorithm is described with referenceto FIG. 3A. An example process for a routing system to determine finalrouting factors between a first capsule layer and a second capsule layerusing a dynamic routing algorithm is described with reference to FIG. 4,The second capsule receives outputs of first capsules of the firstcapsule layer in the receptive field of the second capsule in the firstcapsule layer.

The second capsule determines a respective vote for the second capsulefrom each first capsule of the first capsule layer in the receptivefield of the second capsule (502). The second capsule has a respectiveweight matrix corresponding to each first capsule of the first capsulelayer in the receptive field of the second capsule. To determine a votefor the second capsule from a particular first capsule, the secondcapsule multiplies the output of the particular first capsule by theweight matrix of the second capsule that corresponds to the particularfirst capsule. In some implementations, the output of the particularfirst capsule is a vector. In these implementations, the weight matrixof the second capsule corresponding to the particular first capsule is amatrix with at least two rows and a number of columns that is the sameas the dimensionality of the output of the particular first capsule. Insome implementations, the output of the particular first capsule is amatrix. In these implementations, the weight matrix of the secondcapsule corresponding to the particular first capsule is a matrix withat least two rows and a number of columns that is the same as the numberof rows of the output of the particular first capsule.

In some implementations, When the first capsule layer and the secondcapsule layer are both convolutional capsule layers, the weight matricesof capsules of a second type in the second capsule layer correspondingto capsules of a first type in the first capsule layer are all shared(i.e., the weight matrices are the same). In some of theseimplementations, the system adds the coordinates of the center of thereceptive field of the particular first capsule in the input image tothe vote for the second capsule from the particular first capsule. Insome implementations, the coordinates of the center of the receptivefield of the first capsule are given by the x-y coordinates (in theframe of reference of the input image) of a pixel that is determined tobe the center (e.g., the centroid) of the set of pixels of the inputimage that are in the receptive field of the first capsule. In someimplementations, the coordinates of the center of the receptive field ofthe particular first capsule may be scaled (e.g., such that eachcoordinate is in the range 0 to 1). In some implementations, when thevote for the second capsule from the particular first capsule is amatrix, the system adds the coordinates of the center of the receptivefield of the particular first capsule to the first two components of therightmost column of the vote for the second capsule from the particularfirst capsule (e.g., the x-coordinate of the center of the receptivefield may be added to the first component of the rightmost column of thevote and the y-coordinate may be added to the second component of therightmost column of the vote).

The second capsule determines weights for the votes for the secondcapsule (504). In some implementations, the second capsule determinesthe weight for a vote for the second capsule from a first capsule of thefirst capsule layer based on the value of the final routing factorbetween the first capsule and the second capsule. In some of theseimplementations, the second capsule determines the weight for a vote forthe second capsule from the first capsule of the first capsule layeradditionally based on an activation of the first capsule of the firstcapsule layer.

The second capsule determines the second capsule output by combining theweighted votes for the second capsule (506). In some implementations,the routing system is an expectation-maximization routing system (e.g.,the expectation-maximization routing system described with reference toFIG. 3A), and the second capsule determines the second capsule output asa mean of the weighted votes for the second capsule, as described withreference to 308. In some other implementations, the routing system is adynamic routing system (e.g., the dynamic routing system described withreference to FIG. 4), and the second capsule determines the secondcapsule output as a sum of the weighted votes for the second capsule, asdescribed with reference to 402.

In some implementations, the routing system is anexpectation-maximization routing system (e.g., theexpectation-maximization routing system described with reference to FIG.3A), and the second capsule determines parameters characterizing thedistribution of weighted votes for the second capsule and a costassociated with the second capsule (510), as described with reference to308 and 310. In some of these implementations, the second capsuledetermines an activation of the second capsule (512), as described withreference to 312.

In some other implementations, the routing system is a dynamic routingsystem (e.g., the dynamic routing system described with reference toFIG. 4), and the second capsule normalizes the output of the secondcapsule (514), as described with reference to 404.

This specification uses the term “configured” in connection with systemsand computer program components. For a system of one or more computersto be configured to perform particular operations or actions means thatthe system has installed on it software, firmware, hardware, or acombination of them that in operation cause the system to perform theoperations or actions. For one or more computer programs to beconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. Alternatively or in addition, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can alsobe, or further include, special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application-specificintegrated circuit). The apparatus can optionally include, in additionto hardware, code that creates an execution environment for computerprograms, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them.

A computer program, which may also be referred to or described as aprogram, software, a software application, an app, a module, a softwaremodule, a script, or code, can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages; and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A program may, but neednot, correspond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data, e.g., one or morescripts stored in a markup language document, in a single file dedicatedto the program in question, or in multiple coordinated files, e.g.,files that store one or more modules, sub-programs, or portions of code.A computer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a data communication network.

In this specification the term “engine” is used broadly to refer to asoftware-based system, subsystem, or process that is programmed toperform one or more specific functions. Generally, an engine will beimplemented as one or more software modules or components, installed onone or more computers in one or more locations. In some cases, one ormore computers will be dedicated to a particular engine; in other cases,multiple engines can be installed and running on the same computer orcomputers.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA or an ASIC, or by acombination of special purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for performing or executing instructions and one or morememory devices for storing instructions and data. The central processingunit and the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.Also, a computer can interact with a user by sending text messages orother forms of message to a personal device, e.g., a smartphone that isrunning a messaging application, and receiving responsive messages fromthe user in return.

Data processing apparatus for implementing machine learning models canalso include, for example, special-purpose hardware accelerator unitsfor processing common and compute-intensive parts of machine learningtraining or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machinelearning framework, e.g., a TensorFlow framework, a Microsoft CognitiveToolkit framework, an Apache Singa framework, or an Apache MXNetframework.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited inthe claims in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system modules and components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

1. One or more computers and one or more storage devices storinginstructions that when executed by the one or more computers cause theone or more computers to implement: a neural network that is configuredto receive a network input and to generate a network output for thenetwork input, wherein: the neural network comprises a plurality oflayers arranged in a sequence, including a plurality of capsule layers;each capsule layer comprises a respective plurality of capsules; andeach particular capsule in a particular capsule layer is configured to:receive respective inputs including: (i) outputs generated by capsulesof a previous capsule layer that is before the particular capsule layerin the sequence, and (ii) final routing factors between capsules of theprevious capsule layer and the particular capsule, wherein the finalrouting factors are generated by a routing subsystem; and determine aparticular capsule output based on the received inputs, wherein theparticular capsule output is of dimension greater than one; and arouting subsystem that is configured to perform operations including:receiving outputs generated by capsules in a first capsule layer;determining final routing factors between capsules of the first capsulelayer and capsules of a second capsule layer that is after the firstcapsule layer in the sequence; and providing the final routing factorsto the capsules in the second capsule layer.
 2. The system of claim 1,wherein determining final routing factors between capsules of the firstcapsule layer and capsules of a second capsule layer that is after thefirst capsule layer in the sequence comprises: performing one or moreiterations of an expectation-maximization routing algorithm, whereineach iteration of the expectation-maximization routing algorithmincludes an M-step and an E-step, wherein for a particular iteration:the M-step includes, for each second capsule in the second capsulelayer: determining weighted votes for the second capsule from one ormore first capsules of the first capsule layer for the particulariteration, comprising: determining a vote for the second capsule from aparticular first capsule of the first capsule layer by multiplying anoutput of the particular first capsule by a weight matrix of the secondcapsule that is associated with the particular first capsule; weightingthe vote for the second capsule from the particular first capsule of thefirst capsule layer by a routing factor between the particular firstcapsule and the second capsule for the particular iteration; determiningvalues of one or more parameters characterizing a distribution ofweighted votes for the second capsule from the one or more firstcapsules of the first capsule layer for the particular iteration;determining a cost associated with the second capsule for the particulariteration based at least in part on the values of the parameterscharacterizing the distribution of weighted votes for the second capsulefor the particular iteration; determining an activation of the secondcapsule for the particular iteration based at least in part on the costassociated with the second capsule for the particular iteration; and theE-step includes determining routing factors for a next iteration basedon at least the activations of the capsules of the second capsule layerfor the particular iteration and the values of the parameterscharacterizing the distributions of weighted votes associated with thecapsules of the second capsule layer for the particular iteration;determining final routing factors to be the routing factors determinedin a last iteration of the expectation-maximization routing algorithm.3. The system of claim 2, wherein weighting the vote for the secondcapsule from the particular first capsule of the first capsule layerfurther comprises: weighting the vote for the second capsule from theparticular first capsule by an activation of the particular firstcapsule.
 4. The system of claim 2, wherein the cost associated with thesecond capsule for the particular iteration is further determined basedat least in part on the routing factors between the first capsules ofthe first capsule layer and the second capsule for the particulariteration.
 5. The system of claim 2, wherein the cost associated withthe second capsule for the particular iteration is further determinedbased at least in part on an inverse temperature parameter.
 6. Thesystem of claim 2, wherein the parameters characterizing thedistribution of weighted votes for the second capsule for the particulariteration include: a mean of the weighted votes for the second capsulefrom the one or more first capsules of the first capsule layer for theparticular iteration; and a variance of the weighted votes for thesecond capsule from the one or more first capsules of the first capsulelayer for the particular iteration.
 7. The system of claim 1, whereindetermining final routing factors between capsules of the first capsulelayer and capsules of a second capsule layer that is after the firstcapsule layer in the sequence comprises: performing one or moreiterations of a dynamic routing algorithm, wherein a particulariteration of the dynamic routing algorithm includes: for each secondcapsule of the second capsule layer: determining weighted votes for thesecond capsule from one or more first capsules of the first capsulelayer for the particular iteration, comprising: determining a vote forthe second capsule from a particular first capsule of the first capsulelayer by multiplying the output of the particular first capsule by aweight matrix of the second capsule that is associated with theparticular first capsule; weighting the vote for the second capsule fromthe particular first capsule of the first capsule layer by a normalizedrouting factor between the particular first capsule and the secondcapsule for the particular iteration; determining routing factors for anext iteration, wherein determining a routing factor between a firstcapsule of the first capsule layer and a second capsule of the secondcapsule layer for the next iteration comprises: determining a sum of theweighted votes for the second capsule from the one or more firstcapsules of the first capsule layer for the particular iteration;determining a similarity measure between the sum of the weighted votesfor the second capsule for the particular iteration and the vote for thesecond capsule from the first capsule; determining the routing factorbetween the first capsule and the second capsule for the next iterationby combining the determined similarity measure with the routing factorbetween the first capsule and the second capsule for the particulariteration; determining final routing factors to be the routing factorsdetermined in a last iteration of the dynamic routing algorithm.
 8. Thesystem of claim 7, wherein determining the sum of the weighted votes forthe second capsule further comprises normalizing the sum of the weightedvotes for the second capsule.
 9. The system of claim 7, wherein thesimilarity measure between the sum of the weighted votes for the secondcapsule and the vote for the second capsule from the first capsulecomprises: the inner product of the sum of the weighted votes for thesecond capsule and the vote for the second capsule from the firstcapsule.
 10. The system of claim 2, wherein determining a particularcapsule output based on the received inputs comprises: determiningweighted votes for the particular capsule from the capsules of theprevious capsule layer, comprising: determining a vote for theparticular capsule from a given capsule in the previous capsule layer bymultiplying the output of the given capsule by a weight matrix of theparticular capsule that is associated with the given capsule; weightingthe vote for the particular capsule from the given capsule in theprevious capsule layer by a final routing factor between the givencapsule and the particular capsule; determining the particular capsuleoutput by combining the weighted votes for the particular capsule fromthe capsules of the previous capsule layer.
 11. The system of claim 10,wherein the final routing factors are normalized.
 12. The system ofclaim 10, wherein weighting the vote for the particular capsule from thegiven capsule in the previous capsule layer further comprises: weightingthe vote for the particular capsule from the given capsule by anactivation of the given first capsule.
 13. The system of claim 10,wherein combining the weighted votes for the particular capsule from thecapsules of the previous capsule layer comprises: determining a mean ofthe weighted votes for the particular capsule from the capsules of theprevious capsule layer.
 14. The system of claim 10, wherein determininga particular capsule output based on the received inputs furthercomprises: determining values of one or more parameters characterizing adistribution of the weighted votes for the particular capsule from thecapsules of the previous capsule layer; determining a cost associatedwith the particular capsule based at least in part on the values of theparameters characterizing the distribution of the weighted votes for theparticular capsule; determining an activation of the particular capsulebased at least in part on the cost associated with the particularcapsule.
 15. The system of claim 1, wherein the output of a capsulecomprises an activity vector.
 16. The system of claim 1, wherein theoutput of a capsule comprises an activity matrix.
 17. The system ofclaim 1, wherein the neural network comprises an initial convolutionallayer followed by a primary capsule layer.
 18. The system of claim 17,wherein the primary capsule layer is followed by a plurality ofconvolutional capsule layers.
 19. The system of claim 18, wherein a lastconvolutional capsule layer of the plurality of convolutional capsulelayers is followed by a class capsule layer.
 20. The system of claim 19,wherein: for each capsule of the class capsule layer, weight matrices ofthe capsule that are associated with capsules of the last convolutionalcapsule layer of a same type share weight values, wherein: a firstcapsule of a convolutional capsule layer is of a same type as a secondcapsule of the convolutional capsule layer if weight matrices of thefirst capsule share weight values with weight matrices of the secondcapsule; values based on the coordinates of the center of the receptivefields of capsules of the last convolutional capsule layer are added tovotes of capsules of the last convolutional capsule layer.
 21. Thesystem of claim 19, wherein the neural network is configured to receivea network input and to classify the network input as belonging to one ormore of a predetermined set of classes, and wherein the class capsulelayer includes a respective capsule for each class in the predeterminedset.
 22. The system of claim 21, wherein: an activation of a respectivecapsule of the class capsule layer is greater than activations of othercapsules of the class capsule layer; the network input is classified asbelonging to a particular class corresponding to the respective capsuleof the class capsule layer.
 23. The system of claim 1, wherein thenetwork input is an image.
 24. One or more non-transitory computerstorage media storing instructions that when executed by one or morecomputers cause the one or more computers to implement: a neural networkthat is configured to receive a network input and to generate a networkoutput for the network input, wherein: the neural network comprises aplurality of layers arranged in a sequence, including a plurality ofcapsule layers; each capsule layer comprises a respective plurality ofcapsules; and each particular capsule in a particular capsule layer isconfigured to: receive respective inputs including: (i) outputsgenerated by capsules of a previous capsule layer that is before theparticular capsule layer in the sequence, and (ii) final routing factorsbetween capsules of the previous capsule layer and the particularcapsule, wherein the final routing factors are generated by a routingsubsystem; and determine a particular capsule output based on thereceived inputs, wherein the particular capsule output is of dimensiongreater than one; and a routing subsystem that is configured to performoperations including: receiving outputs generated by capsules in a firstcapsule layer; determining final routing factors between capsules of thefirst capsule layer and capsules of a second capsule layer that is afterthe first capsule layer in the sequence; and providing the final routingfactors to the capsules in the second capsule layer.
 25. A methodperformed by one or more data processing apparatus, the methodcomprising: receiving a network input; and processing the network inputusing a neural network and a routing subsystem, wherein: the neuralnetwork is configured to receive the network input and to generate anetwork output for the network input, wherein: the neural networkcomprises a plurality of layers arranged in a sequence, including aplurality of capsule layers; each capsule layer comprises a respectiveplurality of capsules; and each particular capsule in a particularcapsule layer is configured to: receive respective inputs including: (i)outputs generated by capsules of a previous capsule layer that is beforethe particular capsule layer in the sequence, and (ii) final routingfactors between capsules of the previous capsule layer and theparticular capsule, wherein the final routing factors are generated by arouting subsystem; and determine a particular capsule output based onthe received inputs, wherein the particular capsule output is ofdimension greater than one; and the routing subsystem is configured toperform operations including: receiving outputs generated by capsules ina first capsule layer; determining final routing factors betweencapsules of the first capsule layer and capsules of a second capsulelayer that is after the first capsule layer in the sequence; andproviding the final routing factors to the capsules in the secondcapsule layer.